To solve the tracking drift problem caused by the low discriminability of sample information in typical correlation filtering framework, a correlation filter based object tracking algorithm with spatial structure information was proposed. Firstly, the spatial context structure constraint was introduced to optimize the model construction, meanwhile, the regularized least square and matrix decomposition idea were exploited to achieve the closed solution. Then, the complementary features were used for the target apparent description, and the scale factor pool was utilized to deal with target scale changing. Finally, according to the occlusion influence of target judged by motion continuity, the corresponding model updating strategy was designed. Experimental results demonstrate that compared with the traditional algorithm, the precision of the proposed algorithm is increased by 17.63%, and the success rate is improved by 24.93% in various typical test scenarios, achieving more robust tracking effect.